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ABSTRACT 


Automation and robotics for space applications are being pursued for increased productivity, 
enhanced reliability, increased flexibility, higher safety, and for the automation of time- 
consuming tasks and those activities which are beyond the capacity of the crew. One of the key 
functional elements of an automated robotic system is sensing and perception. As the robotics 
era dawns in space, vision systems will be required to provide the key sensory data needed for 
multifaceted intelligent operations. In general, the three-dimensional scene/object description, 
along with location, orientation, and motion parameters will be needed. In space, the absence of 
diffused lighting due to a lack of atmosphere gives rise to: (a) high dynamic range (10 8 ) of 
scattered sunlight intensities, resulting in very high contrast between shadowed and specular 
portions of the scene; (b) intense specular reflections causing target/scene bloom; and (c) loss of 
portions of the image due to shadowing and presence of stars. Earth, Moon, and other space 
objects in the scene. In this work, developments for combating the adverse effects described 
earlier and for enhancing scene definition are discussed. Both active and passive sensors are 
used. The algorithm for selecting appropriate wavelength, polarization, look angle of vision 
sensors is based on environmental factors as well as the properties of the target/scene which are 
to be perceived. The environment is characterized on the basis of sunlight and other illumination 
incident on the target/scene and the temperature profiles estimated on the basis of the incident 
illumination. The unknown geometrical and physical parameters are then derived from the 
fusion of the active and passive microwave, infrared, laser, and optical data. 

I- Vision for Vision and Remote Sensing 

NASA has identified five strategic enterprises: Mission to Planet Earth (MTPE), Aero- 
nautics, Human Exploration and Development of Space (HEDS), scientific research, and space 
technology. In each of these areas there is a need for remote sensing and vision. For the MTPE 
Program, one major task is the observation of the Earth and its atmosphere in order to provide 
estimation of resources, and sense, monitor, and model the environment. The HEDS program 
deals with space infrastructure including robotic missions and human expeditions and 
settlements. The need for autonomous systems for servicing, maintenance, repairs, docking, 
assembly, planning, monitoring, diagnosis, control, and fault recovery is of paramount 
importance for this enterprise. The NASA scientific missions are aimed at acquiring knowledge 
about the universe, matter, and the process for the evolution of life. Much of the research in 
this area involves remote observations of celestial bodies, phenomena and processes. These 
observations are carried out in various parts of the electromagnetic spectrum. In all the NASA 
strategic enterprises, the need for automation and robotics has been established for or- 
biter/station/satellite servicing, astronaut assistance, equipment transfer, docking and berthing, 
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inspection, remote monitoring, rocket staging, telescience, and assembly of structures and 
systems. In support of all the tasks involving vision and sensing, technologies will be required 
to assure the perception of objects, scenes, and phenomena in space. In general, key sensing 
data will be needed for the three-dimensional scene/object description, including location, 
orientation, motion parameters, and surface/subsurface properties. The need and status for the 
vision technology applied to robotics and automation have been research by NASA over the 
past decade and have been summarized by Krishen l 1 * 2 !. For the remote sensing applications, 
the processed sensor data should provide surface properties such as roughness, dielectric 
constant, emissivity, reflectivity, temperature, orientation, and slope. 

The ultimate goal of vision and sensing systems is to be able to acquire required scene/object pa- 
rameters at any time, location, or illumination condition. This assured vision has been the 
subject of fusion by Collin, Krishen, and Pampagninl 3 !. The possibility of this happening rests 
on understanding the reflectance, emission/absorption, and scattering properties of 
objects/scenes based on physical interaction of the illuminations and the material composition of 
the object/scene. Once this is approximately realized thorough mathematical and empirical 
methods, it is then applied to identifying sensing strategies. For example, the environment is 
monitored in terms of the radiation received form the sun or any other source. Based on this 
known radiation environment, the scattered and emitted radiation in various parts of the 
electromagnetic (EM) spectrum can be estimated. Then a selection of the sensor parameters can 
be accomplished and vision data acquired. The data can be used to provide the needed 
parameters using physical and phenomenological models. One key element of this approach is 
to use active and passive sensors located in various parts of the EM spectrum. The combi- 
nation of needed sensors and modes for these sensors will depend on the object/scene para- 
meters and the accuracy to which they need to be determined. In this paper, a scheme for new 
multisensor approach to sensing and vision will be described. 

n. Qb je ct / S c en e P ar ame ter E stimatio n 

The natural space environment consists of intense light and dark period. At a nominal 
altitude of 270 nmi, the sunlight intensity will fluctuate between about 60 minutes of extreme 
brightness (13,000 ft-c) and 30 minutes of darkness. Furthermore, due to the absence of 
atmosphere, light is not diffused/scattered. Consequently, the unenhanced images have large 
contrast with intensity changes of the order of 10. The intense specular reflections combined 
with camera performance can cause bloom and Faunhofer/Airy rings resulting in scene obscurity. 
Further complexity results from other objects, such as stars, moon, sun. Earth, and other 
satellites in the field of view (FOV). Object reflectivity can also pose problems for the vision 
systems. Most space systems are painted white or finished with smooth, specular materials to 
provide highly reflective surfaces. The ubiquity of white surfaces intensifies the problem of 
relying on photometric data for object identification/discrimination. A secondary source of 
concern affecting vision is the absence of gravity. For free-flying and tethered objects there 
would be an increased number of positions and orientations in which the objects may be found 
due to the lack of disturbances caused by aerodynamic and gravitational forces. 
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There is also a wide range of temperatures from a few degrees Kelvin to hundreds of degrees 
exhibited by objects and scenes. The surface reflectance can range from 0.07 for lunar soil and 
thermal protection to 1 for polished metallic surfaces. 

A crucial step in the estimation of object/scene parameters is the monitoring and intelligent use 
of environmental information in space. The spacecraft and other space objects to be recognized 
by a robot have a definite geometry and consist of materials manufactured here on the Earth. 
Furthermore, their orbits and general orientations can also be initially available or estimated. For 
the lunar or Mars surface, there is similar data available in terms of surface orientation as a 
function of sun and Earth location or time. With this type of information available from objects 
and scenes in space, one can develop the surface illumination intensity pattern using 
computerized model of the sun radiation intensity. This can also be extended to provide initial 
temperature and emissivity estimates using Planck's law and the Rayleigh-Jeans approximations. 
Figure 1 illustrates the initially estimated data from orbital and illumination considerations. 

The second step in the approach is to calculate scattering and emission patterns based on known 
geometry and orbital parameters. For the lunar surface these calculations can be done on the 
basis of surface height and slope distributions. Once the scattering and emissions are estimated, 
the selection of active and passive sensors can be effectively done. The sensors include 
microwave, millimeter wave, laser, infrared, and optical types. 

The operating frequencies, polarization and look angles for these sensors can also be selected for 
the object/scene viewing. This selection provides the initial estimation of the required 
parameters. The laser scanner can be utilized as an initial estimate for the velocity and 
orientation of the object. This can be followed by another sensor such as an active microwave 
radar to estimate the roughness of the scene/object. Once some roughness estimate is made, a 
radiometer can be used to estimate the dielectric constant. The dielectric constant, along with 
the initial value of the incident radiation can then be utilized to select appropriate infrared 
sensor to map the scene/object. In the last step, if the optical image is available, the data can be 
fused to provide the synergism needed to refine the estimates of roughness, dielectric properties, 
and temperatures. The overall flow is briefly shown in Figure 2. 

The fusion of data including sensor information takes into account physical scattering and 
emissivity models and estimates more accurately the object/scene parameters and/or provides 
new parameters that could not be estimated by the sensors on an individual basis. For example, 
the fusion of active and passive microwave data could lead to the estimation of roughness, 
dielectric properties, and the root mean square height distribution. 

m. Status of the Sensor Fusion 

The initial work on sensor fusion as described in this paper was carried out by 
identifying objects, parts of which are occluded, shadowed, or wiped by intense specular 
reflection. This work resulted in a patent! 4 !. The algorithm developed calculates radar scattering 
cross sections (RCS) for the visible portion of the object. Then a minimization technique is 
used to compare measured RCS with the calculated ones. The object geometry is then perturbed 
to minimize the difference between measured and calculated values. This procedure leads to a 


443 



Intelligent Environment 
Estimations 


» Specularities, shadows, 

illumination intensity maps 


• Temperatures 

• Known surface properties 

y Object Orbit and Location w.r.t 
Sun/Moon/Earth 



Figure 1. Estimated Environment 



Figure 2. Sensor Fusion Results 


444 




better geometrical description of the object. This work was then generalized for the lunar 
outpost geometry using multisensorsl 5 > 6 l. An algorithm was developed based on the general 
discussion presented in Section II of this paper. The description of the object/scene is provided 
by fusing the appropriate data using physical models. Fuzzy description models are used to 
alleviate the difficulty of collecting highly-calibrated data. So that the responses of the sensors 
are identified as very low, low, medium, high, very high, etc., as opposed to numbers. This 
approach has yielded results using simulated lunar scenes. A spatial map of the needed 
parameters can be generated through this scheme! 6 ]. 

IV. Concluding Remarks 

The sensor fusion described in this paper advances the state of the art by using unique 
algorithms based on physical models of scattering and emission from space objects and scenes. 
The ultimate objective is to be able to switch sensors automatically according to the parameters 
that need to be estimated through observation. Currently, these models are being refined at JSC 
and experimental verification plans being developed for the multisensor fusion using laser, 
optical, visible, infrared, microwave, and radiometer data. 
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